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Abstract 

In order to estimate the likely effects on item difficulty when a calculator becomes available on 
the quantitative section of the Graduate Record Examinations® (GRE®-Q), 168 items (in six 28- 
item forms) were administered either with or without access to an on-screen four-function 
calculator. The forms were administered as a special research section at the end of operational 
tests, with student volunteers randomly assigned to the calculator or no-calculator groups. Usable 
data were obtained from 13,159 participants. Test development specialists were asked to rate 
which items they thought would become easier with a calculator. In general, the specialists were 
successful in identifying the items with relatively large calculator effects, though even these 
effects were quite small. An increase of only about four points in the percent correct should 
suffice for the items identified as likely to show calculator effects with no adjustment needed for 
the majority of the items. Introduction of a calculator should have little or no effect on gender 
and ethnic differences. 

Key words: Item difficulty, quantitative tests, reasoning skills, computational errors 
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Professional standards for assessing quantitative reasoning skills suggest that calculators 
should be provided to examinees. Because the quantitative portion of the Graduate Record 
Examinations® General Test (GRE -Q) emphasizes reasoning skills, and not computational 
facility, no complex computations are required and a calculator is not needed. Nevertheless, 
providing a calculator helps to assure that trivial computational errors are not interfering with 
assessment of the intended reasoning construct. Despite these advantages, introduction of a 
calculator also introduces some challenges. Specifically, item difficulty estimates for items 
pretested without a calculator will, in some cases, change when a calculator is introduced. If the 
effect of the calculator can be estimated for various item types, then the existing item difficulty 
indices can be appropriately adjusted. 

Research with a number of different tests has demonstrated the potential effect of 
calculators on item difficulty (Ansley, Spratt, & Forsyth, 1989; Heam & Loyd, 1987; Loyd, 

1991; Morgan & Stevens, 1991). A large-scale (7,000 examinees from 275 high schools) study 
conducted when the calculator was introduced for the SAT®-I indicated a generally positive 
effect (items became easier) for students who were allowed to use calculators (Bridgeman, 
Harvey, & Braswell, 1995). Nevertheless, on individual items the calculator could have either no 
effect on item difficulty, a positive effect, or a negative effect. Items that required nontrivial 
calculations tended to show positive effects. Only 5 items out of 70 showed negative effects; an 
example of such an item asked for the remainder when 63,383 is divided by 7. Test developers 
were reasonably successful in predicting which items would show calculator effects, but they 
also made some errors. Test developers correctly predicted 20 of the 37 items that showed 
substantially positive effects; 4 items that were predicted to show negative calculator effects 
actually showed modest positive effects. In a subsequent unpublished study, I. Lawrence 
(personal communication, August 2003) developed an adjustment factor so that the difficulty of 
items of particular types that were pretested without a calculator could be estimated when they 
were administered with a calculator. This was not a very precise adjustment, but precision was 
not necessary, since all SAT-I tests are equated after the administration, as will be done for the 
new GRE linear forms that will replace the GRE CAT in 2006. The adjustment did, nevertheless, 
help to ensure that the early fonns for the test with calculators allowed would meet the 
appropriate difficulty specifications. 
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In addition to identifying the adjustment factor for the GRE that should be applied to 
items that were pretested without a calculator, a second purpose of this research was to evaluate 
the effects of calculator availability on overall scores. In particular, we sought to determine 
whether introducing a calculator would have any differential impact on gender and ethnic 
groups. 


Procedure 

Test Forms 

From the pool of disclosed items that are not widely used in test-preparation materials 
(e.g., items from forms released in accordance with New York state requirements), we assembled 
six 28-item linear tests that roughly met the specifications proposed for the new test, for a total of 
168 items. We created forms that are consistent with the content changes being recommended for 
the revised GRE-Q. Specifically, the test forms reflect an increased emphasis on items classified 
as real, less emphasis on geometry items, and a slightly higher proportion of data-interpretation 
items. Because our goal was to estimate changes in difficulty for existing types for which 
difficulty had already been estimated in a no-calculator format, we did not include any of the 
new format items. These new-format items are being developed and tested in a calculator- 
available mode. 

Test development staff screened the pool of items for those items that become trivial 
when examinees have access to a calculator. Test development staff also rated each item for 
calculator sensitivity (negative effect, positive effect, or no effect). (No effect was originally two 
different points on the scale—“item difficulty unlikely to change” and “item does not lend itself 
to using the calculator.” This distinction was not particularly useful, so these categories were 
combined for most analyses.) Three staff members independently rated the items, and then a 
consensus rating was reached after discussion. 

Design 

Each of the six forms was administered with and without calculator, for a total of 12 
groups. Examinees were randomly assigned to one of these groups. For students in the calculator 
condition, an on-screen calculator could be turned on for any item. The calculator was the four- 
function plus square root key calculator that is currently used for the Praxis™ computer-based 
test. 
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Participants 

At the end of the regular GRE tests administered in September and October of 2003, a 
screen appeared that invited voluntary participation in a research project and offered an incentive 
not only to participate, but to perform well on this research section. Specifically, potential 
participants were told that “it is important for our research that you try to do your best on this 
section,” and that they would be eligible to win a $250 prize that would be given “to those 100 
test takers who score the highest on questions in the research section relative to how well they 
did on the preceding scored sections.” 

A total of 15,811 volunteers participated in the study. From this sample, we dropped 9 
who had no valid GRE-Q score on the operational section, and dropped an additional 2,232 who 
did not appear to be taking the experimental section seriously, as evidenced by failing to 
complete three fourths of the section. For the remaining students, we performed an 
equipercentile equating for each of the six forms with the operational quantitative score, and 
dropped 411 students who were likely not trying because they scored 200 or more points lower 
on the experimental section than on the operational section. Thus, we were fairly confident that 
most examinees included in the final sample were making a sincere effort on the experimental 
section. Losses from this sample screening were comparable across the conditions; 1,323 
(17.2%) were dropped from the no-calculator group and 1,329 (16.4%) were dropped from the 
calculator group. The final sample contained 13,159 participants, or about 1,100 for each of the 
12 groups. 


Results 

Individual results for all 168 items are presented in Appendix A. Items are ordered by the 
extent of calculator use in the group that had access to a calculator. For the first 20 items listed, 
at least half of the students who could use a calculator actually did, while for the last 20 items no 
more than 2% of examinees who could use a calculator chose to use it on these items. In order to 
summarize these results, we computed the percentage correct for students who took the item with 
a calculator available and the percentage correct for students who did not have a calculator 
available, and computed the difference. These difference scores ranged from -5 (i.e., percentage 
correct was 5 points higher in the no-calculator condition than in the calculator-available 
condition) to 15. A cross-tabulation of these differences with the test developer ratings of 
calculator sensitivity is provided in Table 1. 
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Table 1 


Cross-Tabulation of Differences in Percentage Correct, With Test Developer Ratings of 

Calculator Impact 

% correct in 

Rating of calculator impact 

Total 

calculator- 

available group 

minus % correct 

in no-calculator 

group 

-1 

0 

1 

2 

-5 

0 

2 

0 

0 

2 

-4 

0 

1 

1 

1 

3 

-3 

0 

7 

3 

3 

13 

-2 

0 

4 

6 

3 

13 

-1 

0 

6 

11 

7 

24 

0 

0 

6 

13 

5 

24 

1 

2 

6 

8 

7 

23 

2 

0 

4 

12 

9 

25 

3 

1 

0 

1 

5 

7 

4 

1 

0 

3 

7 

11 

5 

0 

0 

2 

6 

8 

6 

0 

0 

0 

4 

4 

7 

0 

0 

0 

2 

2 

8 

0 

0 

0 

4 

4 

9 

0 

0 

0 

1 

1 

10 

0 

0 

0 

2 

2 

11 

0 

0 

0 

1 

1 

15 

0 

0 

0 

1 

1 

Total 

4 

36 

60 

68 

168 
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Focusing first on the total column on the far right in Table 1, note that the percentage 
correct in calculator and no-calculator groups was virtually identical (within two percentage 
points) for 109 of the 168 items (bold in table). Only 15 items showed differences of more than 
five percentage points (italics in table). The test developers had identified only four items as 
likely to show a negative impact of calculator availability, but the actual impact was slightly 
positive for all of these items. As noted previously, the distinction between a rating of 0 and 1 
was not meaningful. Therefore, for subsequent analyses we simplified the rating variable into 
just two categories, 0 and 1, with 1, 0, and 1 converted to 0, and 2 converted to 1. This simplified 
variable is used in Figure 1, which shows the scatterplot of percentage correct for items in the 
no-calculator (x-axis) and calculator-available groups. Items that were identified by the test 
development experts as likely to show a calculator effect are signified by + on the graph, while 
the other items are •. The correlation of item difficulties in the two conditions was very high (r = 
.98). Items rated as likely to show a calculator effect were generally above the 45-degree line in 
Figure 1, confirming that these items tend to be somewhat easier when a calculator is available. 
In addition, effects seem to be greatest for middle-difficulty items. Because most examinees get 
easy items right without a calculator, the advantage of having a calculator for these items is 
trivial. Very difficult items are typically conceptually difficult, not computationally difficult, so a 
calculator is of little benefit on these items also. Even in the middle-difficulty range, most items 
show little or no calculator effect. Appendix B presents three specific items that illustrate items 
showing different calculator effects. The first two items are clearly calculation-intensive and 
show a large effect. The third item also requires calculation, but the calculation is so 
straightforward that the calculator is of little benefit. 

Figure 2 shows the scatterplot of percentage correct in the two calculator conditions by 
item type. There are two item types: standard multiple-choice questions with five options (5- 
choice) and quantitative comparison (QC) items in which a quantity in column A must be 
compared to a quantity in column B, with answer choices indicating whether A is larger, B is 
larger, they are equal, or there is insufficient infonnation to decide. 

QC items are generally designed to be answered quickly, with relatively little calculation 
needed. Thus, a calculator would generally not be expected to be very useful on this question 
type, and this seems to be the case, since only 2 of the 78 QC items showed a calculator effect of 
more than 5 percentage points. 
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Calculator Rating 
0-1 scale 

+1 


Figure 1. Scatterplot and regression line showing relationship of item difficulties in the no 
calculator and calculator-available groups. 



Figure 2. Scatterplot and regression line showing relationship of item difficulties in the no 
calculator and calculator-available groups for two item types. 
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Table 2 presents mean differences between percentage correct 1 in the no-calculator and 
calculator-available groups across calculator conditions for a crossing of test developer 
calculator ratings, item type, and pure versus real item classification. In general, real problems 
are word problems with applied contexts, while pure problems are numbers presented with a 
minimum of words. The first row in this table indicates that for the 24 pure, five-choice 
questions that were rated as likely to have no calculator impact, the mean difference between the 
percentage correct in the calculator and no-calculator conditions was 0.21. Across item types, 
items rated as not likely to show calculator effects did, indeed, show minimal mean differences. 
Items rated as likely to show some calculator effects typically showed average differences of 
about 3 or 4 percentage points between the calculator and no-calculator conditions. The eight 
real QC items were an exception, showing an average difference of only 0.25 percentage points 
even when rated as likely to show calculator effects. 

Table 2 

Mean Differences in Item Difficulty (Percentage Correct) for Items of Different Types and 


Rated Sensitivity 

Calc, rating 


Type 

Mean 

N 

SD 

0 

5C 

Pure 

.21 

24 

2.06 



Real 

-.06 

16 

1.95 



Total 

.10 

40 

2.00 


QC 

Pure 

-.33 

48 

2.14 



Real 

.58 

12 

2.35 



Total 

-.15 

60 

2.19 

1 

5C 

Pure 

4.29 

7 

4.42 



Real 

3.26 

43 

3.95 



Total 

3.40 

50 

3.98 


QC 

Pure 

2.50 

10 

2.06 



Real 

.25 

8 

2.19 



Total 

1.50 

18 

2.87 


Note. N is the number of items. Each item was answered by at least 1,000 people in the 
calculator-available group and 1,000 in the no-calculator group. 
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Effects on Total Score 

Given that only 15 out of the 168 items showed calculator effects of more than five 
percentage points, the effect on total scores would be expected to be rather modest. Table 3 
contrasts the total scores (on the 200-800 GRE scale) of examinees in the no-calculator and 
calculator-available conditions. 

Table 3 


Mean Score on Experimental Section for No-Calculator and Calculator-Available Groups 



Mean 


SD 

Sample 

size 

Ethnic 

group/gender 

No¬ 

calculator 

Calculator 

available 

No¬ 

calculator 

Calculator 

available 

No¬ 

calculator 

Calculator 

available 

White/male 

648 

656 

121 

120 

1,214 

1,348 

White/female 

571 

582 

126 

128 

2,226 

2,451 

Asian American/ 
male 

692 

707 

106 

97 

218 

213 

Asian American/ 
female 

642 

641 

123 

116 

233 

265 

African 

American/male 

524 

541 

153 

143 

104 

91 

African 

American/female 

453 

460 

130 

125 

266 

244 

Hispanic/male 

590 

612 

135 

131 

107 

91 

Hispanic/female 

518 

517 

131 

117 

158 

181 

Other/male 

637 

654 

123 

116 

174 

183 

Other/female 

570 

582 

137 

134 

157 

162 

Total/male 

642 

654 

128 

123 

1,817 

1,926 

Total/female 

563 

574 

133 

132 

3,040 

3,303 


As shown in Table 4, an analysis of variance, with the operational quantitative score, 
calculator availability, gender, and ethnic group (African American, Asian American, Hispanic, 
White, and Other) as independent variables, indicated a small but statistically significant main 
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effect for calculator use, but no significant interactions. This analysis treated each of the six 
equated test forms as equivalent in order to create a single dependent variable. Repeating the 
analysis separately for each test form also indicated no significant interactions. This suggests that 
introduction of a calculator should not be expected to have much impact on gender or ethnic 
differences. 

Table 4 


Analysis of Variance for Combined Test Forms 


Source 

Type III sum of 
squares 

Df 

Mean square 

F 

Sig. 

Gender (G) 

437.642 

1 

437.642 

.073 

.787 

Ethnic group (E) 

37,283.443 

4 

9,320.861 

1.557 

.183 

Calculator 

3,3479.804 

1 

33,479.804 

5.591 

.018 

availability (A) 






GRE quantitative 

38,438,106.590 

1 

38,438,106.59 

6,419.47 

.000 

score (Q) 
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GXE 

29,169.979 

4 

7,292.495 

1.218 

.301 

GX A 

758.230 

1 

758.230 

.127 

.722 

GXQ 

14,213.684 

1 

14,213.684 

2.374 

.123 

EX A 

21,590.202 

4 

5,397.551 

.901 

.462 

EXQ 

41,722.743 

4 

10,430.686 

1.742 

.138 

AX Q 

11,763.833 

1 

11,763.833 

1.965 

.161 

GXEX A 

33,801.558 

4 

8,450.389 

1.411 

.227 

GXEXQ 

35,202.048 

4 

8,800.512 

1.470 

.208 

GX AXQ 

1,776.463 

1 

1,776.463 

.297 

.586 

EX AX Q 

17,693.180 

4 

4,423.295 

.739 

.565 

GXEXAXQ 

36,496.132 

4 

9,124.033 

1.524 

.192 

Error 

6,015,2794.346 

10,046 

5,987.736 



Total 

3,796,018,700.00 

0 

10,086 




Corrected total 

185,920,647.600 

10,085 
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Difficulty Differences by Calculator Use 

The above analyses contrast the item difficulties for the groups of students who did or did 
not have access to a calculator during the test. This is the question of primary interest for 
estimating the likely effects on item difficulty of introducing a calculator. A secondary question 
is a comparison of difficulty differences for students who not only had access to a calculator but 
who actually used it for a given question compared to students who had access but chose to 
answer the question without using the calculator. (Used is our shorthand for indicating that the 
examinee switched on the calculator for a particular item; it is possible that for some items the 
calculator was turned on, but that the examinee did not actually do any calculations with it.) At 
the individual item level, calculator use was relatively rare. For 86 of the 168 items, fewer than 
20% of the examinees who could use the calculator actually did use the calculator. For only 20 
items was the calculator used by more than half of the examinees, and the item with the most 
calculator use still had only 61% of the examinees who could use a calculator actually choosing 
to use a calculator on that item. Nevertheless, differences in percentage correct were sometimes 
quite substantial between students who chose to use or not use the available calculator. The 
largest apparent benefit of the calculator was noted for an item on which 36% of the examinees 
chose to use a calculator, and the percentage correct for these examinees was 71%, compared to 
27% for the examinees who chose not to use the calculator. These results must be interpreted 
cautiously because the students choosing to use a calculator also had higher quantitative ability 
as indexed by their average scores from the operational section of the test (657, vs. 567 for 
students who chose not to use a calculator). However, there were a few items in which the mean 
operational scores were higher in the group choosing not to use a calculator, but there was still an 
apparent advantage to calculator use. For an item on which 53% of examinees chose to use a 
calculator, the mean quantitative score was 611 in the not-used group and 593 in the used group, 
but the percentage correct was 66% in the not-used group and 84% in the used group. 

Differences in percentage correct for students choosing to use or not use an available 
calculator are presented in Figure 3. In order to avoid over-interpretation of differences based on 
very small samples (i.e., for items on which very few people chose to use a calculator), Figure 3 
includes only the 20 items for which at least 50% of the students with an available calculator 
chose to use it. Recall that, for these items, at least 39% of the examinees chose not to use a 
calculator. Points above the 45-degree diagonal indicate higher scores in the group that chose to 
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use a calculator. For half of these items, the difference in percentage correct was over 20 
percentage points. Although this result is intriguing, it would not be correct to label it as a true 
calculator effect because of the self-selection in choosing to use a calculator. For items that 
appear to be calculator-sensitive, better students may chose to use a calculator even though they 
might have done just as well without a calculator. 



Figure 3. Scatterplot, regression line (light), and 45-degree line (bold) showing relationship 
of item difficulties in the calculator-available-but-not-used group and the calculator- 
available-and-used group. 

Calculator Effects on Item Times 

For each item, the mean time to complete the item was computed separately for the no¬ 
calculator and calculator-available groups. These times are summarized in Figure 4. 

Points below the 45-degree line indicate items that were answered more quickly in the 
calculator-available group. In general, times appear to be faster for examinees who had access to 
a calculator, especially on the items that were rated as calculator sensitive by the test 
development experts. 
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200-1 



Item Time (seconds) in No-Calculator Group 


Figure 4. Scatterplot, regression line (light), and 45-degree line (bold) showing relationship 
of item time (in seconds) in the no-calculator and calculator-available groups. 


The time advantage for the 20 items with heaviest calculator use is shown in Figure 5. 
The points are predominantly below the 45-degree line, indicating a time advantage to calculator 
use on the most calculator-active items. 


Conclusions 

For most of the GRE-Q items studied, the effect of having access to a calculator was 
relatively small. Although there were a few exceptions, test developers were generally quite 
accurate in identifying which items were likely to show substantial calculator effects. For items 
identified by test developers as likely to show effects, adding about four percentage points to the 
existing difficulty estimate should suffice. An exception to this rule could be made for real QC 
items, which do not appear to need an adjustment even when identified by test developers as 
likely to show a calculator effect. Because test fonns will be equated after they are administered, 
the difficulty estimates do not have to be nearly as precise as they would for the pre-calibrated 
items used in the current computer-adaptive test. These results suggest that the adjusted estimates 
should be good enough for a test with post-administration equating. The few items that show 
substantial calculator effects should not interfere with the ability of the test as a whole to provide 
comparable reported scores. 
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Figure 5. Scatterplot and 45-degree line showing relationship of item time in the no¬ 
calculator and calculator-available groups for the 20 items with the heaviest calculator use. 

Note. Includes only the 20 items for which at least 50% of examinees chose to use a calculator. 

The substantial effects noted for examinees who choose to use a calculator, compared to 
those who choose not to use it when available, are open to different interpretations and do not 
necessarily reflect a true calculator effect. Nevertheless, they suggest that continued monitoring 
is desirable as test takers become more familiar with ways to use the calculator most effectively. 

Any time differences related to calculator use should not be of great concern. The time 
limits for the new GRE are being set in field trials that include access to a calculator, so these 
time differences are already being taken into account. 

Calculator benefits appeared to be relatively constant across gender and ethnic groups, 
with no significant interactions of gender or ethnicity with calculator availability. As coaching 
schools and tutors begin to prepare students for a calculator-available GRE, this situation could 
change. The best way to prevent any disparate impact from the introduction of calculators is to 
make sure that materials that demonstrate the most effective ways to use a calculator on the GRE 
are available to all students. 
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Notes 

1 Computing means on differences in percentage correct can be problematic because differences 
at the extremes (e. g., 90% to 95% correct) probably reflect greater differences in intrinsic 
difficulty than differences near the middle (e.g., 40% to 45% correct). To adjust for this, a 
nonlinear transformation of percent correct to z-scores or delta scores sometimes is used. 
However, this transformation makes a noticeable difference only for extreme values (Donlon 
& Livingston, 1984). Over the range of difficulties in the current study, the transformation is 
virtually linear, so we used the simpler percent correct metric. 
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Appendix A 

Item Classification, Percentage Correct, and Item Time for Examinees With and Without Calculator Availability, and 

Calculator Usage for Each Item 


Item classification No calculator Calculator available 


Item 

number 

Accession 

number 

Type 

1 

2 

Subject 

Developer 

rating 

% 

correct 

Time 

% 

correct 

Time 

% 

used 

1 

DQ001343 

QC 

Arithmetic 

Pure 

Measurement conversion 

2 

56% 

71 

64% 

61 

61% 

2 

VB3 62260 

5C 

Arithmetic 

Real 

Interest, sales tax, etc. 

2 

80% 

131 

85% 

121 

61% 

3 

DQ002624 

5C 

Data 

Real 

Percent (basic) 

2 

78% 

84 

84% 

72 

60% 

4 

VB368883 

5C 

Geometry 

Real 

Quadrilateral area 

2 

60% 

146 

66% 

112 

59% 

5 

DQ002625 

5C 

Data 

Real 

Percentage change 

2 

49% 

98 

64% 

89 

57% 

6 

LB006142 

5C 

Data 

Pure 

Mean (arithmetic mean) 

2 

79% 

97 

81% 

98 

57% 

7 

WV000485 

5C 

Arithmetic 

Real 

Ratio, proportion 

2 

80% 

64 

82% 

54 

57% 

8 

DM100263 

5C 

Data 

Real 

Weighted mean 

2 

75% 

85 

79% 

80 

56% 

9 

LB001461 

5C 

Data 

Real 

Percentage change 

2 

43% 

93 

54% 

94 

55% 

10 

LB007148 

5C 

Arithmetic 

Real 

Rate (e.g., MPG, dollars/hour) 

2 

82% 

65 

84% 

57 

55% 

11 

DQ003334 

5C 

Data 

Real 

Computation—decimals 

2 

41% 

125 

46% 

121 

54% 

12 

VB356082 

5C 

Data 

Real 

Sets 

2 

75% 

112 

75% 

113 

53% 

13 

DQ002533 

5C 

Data 

Real 

Rate 

2 

68% 

127 

72% 

102 

53% 

14 

LB007115 

QC 

Data 

Real 

Computation-fractions 

2 

80% 

93 

84% 

91 

53% 
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Table (continued) 


Item classification No calculator Calculator available 


Item 

number 

Accession 

number 

Type 

1 

2 

Subject 

Developer 

rating 

% 

correct 

Time 

% 

correct 

Time 

% 

used 

15 

DQ003332 

5C 

Data 

Real 

Computation-fractions 

2 

64% 

112 

72% 

98 

52% 

16 

LB001489 

5C 

Data 

Real 

Computation—decimals 

2 

69% 

83 

74% 

67 

52% 

17 

LB008809 

5C 

Data 

Real 

Counting (combinatorics) 

2 

42% 

83 

47% 

77 

51% 

18 

WW001574 

5C 

Data 

Real 

Probability 

2 

73% 

89 

77% 

80 

51% 

19 

LB001671 

5C 

Arithmetic 

Pure 

Factors, multiples, divisibility 

2 

63% 

84 

73% 

72 

50% 

20 

WW001575 

5C 

Data 

Real 

Mean (arithmetic mean) 

2 

69% 

66 

77% 

56 

50% 

21 

M-072094 

5C 

Arithmetic 

Real 

Ratio, proportion 

2 

75% 

102 

83% 

93 

50% 

22 

VB179876 

5C 

Algebra 

Real 

Linear equation 

2 

77% 

90 

79% 

87 

46% 

23 

VB102042 

5C 

Geometry 

Pure 

Combination 

2 

37% 

90 

38% 

84 

46% 

24 

DQ007280 

5C 

Data 

Real 

Percent (basic) 

2 

74% 

65 

75% 

59 

46% 

25 

DR000032 

5C 

Data 

Real 

Percent (basic) 

2 

67% 

119 

65% 

116 

45% 

26 

VB371114 

5C 

Arithmetic 

Real 

Profit and loss 

2 

61% 

148 

67% 

134 

45% 

27 

AY002038 

5C 

Data 

Pure 

Mean (arithmetic mean) 

1 

64% 

105 

65% 

95 

45% 

28 

DQ003246 

5C 

Algebra 

Pure 

Linear equation 

2 

67% 

112 

73% 

105 

44% 

29 

LB022229 

5C 

Data 

Real 

Estimation 

2 

43% 

184 

47% 

187 

44% 


(Table continues) 




Table (continued) 


Item classification No calculator Calculator available 


Item 

number 

Accession 

number 

Type 

1 

2 

Subject 

Developer 

rating 

% 

correct 

Time 

% 

correct 

Time 

% 

used 

30 

LB001956 

QC 

Data 

Pure 

Mean (arithmetic mean) 

2 

74% 

52 

77% 

47 

43% 

31 

DM001716 

5C 

Data 

Real 

Percent change 

2 

33% 

70 

31% 

65 

43% 

32 

DQ003794 

5C 

Data 

Real 

Percent (basic) 

2 

73% 

98 

69% 

93 

42% 

33 

LBO10341 

5C 

Data 

Real 

Probability 

2 

66% 

121 

69% 

114 

42% 

34 

DW001378 

5C 

Arithmetic 

Real 

Ratio, proportion 

2 

71% 

108 

74% 

103 

41% 

35 

WW001961 

QC 

Data 

Real 

Weighted mean 

1 

51% 

70 

53% 

67 

41% 

36 

DQ002656 

5C 

Algebra 

Pure 

Linear equation 

1 

84% 

69 

83% 

62 

41% 

37 

LB002851 

5C 

Geometry 

Real 

Pythagorean theorem 

2 

63% 

86 

70% 

75 

40% 

38 

DM001718 

5C 

Data 

Real 

Percent (basic) 

2 

33% 

92 

38% 

80 

40% 

39 

DC005107 

5C 

Data 

Real 

Mean (arithmetic mean) 

2 

84% 

49 

83% 

48 

39% 

40 

VB376016 

5C 

Algebra 

Pure 

Quadratic/other/inequality 

1 

65% 

76 

66% 

78 

39% 

41 

DQ001367 

5C 

Data 

Real 

Probability 

1 

67% 

47 

67% 

46 

39% 

42 

DQ002536 

5C 

Data 

Real 

Ratio, proportion 

2 

54% 

110 

59% 

110 

39% 

43 

VB175951 

5C 

Algebra 

Pure 

Applying formula 

2 

89% 

53 

88% 

54 

38% 

44 

DQ003796 

5C 

Data 

Real 

Angles in the plane 

2 

59% 

71 

62% 

66 

38% 


(Table continues) 



Table (continued) 


Item classification No calculator Calculator available 


Item 

number 

Accession 

number 

Type 

1 

2 

Subject 

Developer 

rating 

% 

correct 

Time 

% 

correct 

Time 

% 

used 

45 

DM002194 

5C 

Data 

Real 

Measurement conversion 

1 

73% 

79 

72% 

76 

37% 

46 

DR000328 

5C 

Geometry 

Real 

Quadrilateral perimeter 

1 

74% 

75 

78% 

72 

37% 

47 

LBO10440 

5C 

Data 

Real 

Mean (arithmetic mean) 

2 

57% 

123 

59% 

122 

37% 

48 

LB001459 

5C 

Data 

Real 

How many categories 

2 

34% 

77 

43% 

77 

36% 

49 

DW002095 

5C 

Data 

Real 

Percent of Percent6 

2 

23% 

131 

24% 

130 

36% 

50 

LB022113 

QC 

Algebra 

Real 

Profit and loss 

-1 

37% 

70 

40% 

69 

35% 

51 

PV000319 

QC 

Arithmetic 

Real 

Percent (basic) 

2 

83% 

81 

84% 

76 

35% 

52 

DM001305 

5C 

Data 

Pure 

Probability 

1 

41% 

71 

46% 

71 

35% 

53 

VB366166 

5C 

Arithmetic 

Pure 

Negative exponents 

2 

62% 

48 

72% 

47 

34% 

54 

M-069053 

QC 

Algebra 

Pure 

Linear equation 

1 

79% 

40 

82% 

41 

33% 

55 

DQ001368 

5C 

Data 

Real 

Median 

0 

59% 

66 

57% 

65 

33% 

56 

DR000035 

5C 

Data 

Real 

Mean (arithmetic mean) 

1 

52% 

75 

54% 

94 

33% 

57 

DC000493 

5C 

Data 

Real 

Percent (basic) 

2 

84% 

62 

81% 

63 

33% 

58 

VB380644 

QC 

Arithmetic 

Pure 

Exponents 

1 

19% 

61 

19% 

60 

32% 

59 

DM002418 

QC 

Algebra 

Pure 

Newly defined functions 

2 

66% 

75 

66% 

83 

30% 
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Table (continued) 


Item classification No calculator Calculator available 


Item 

number 

Accession 

number 

Type 

1 

2 

Subject 

Developer 

rating 

% 

correct 

Time 

% 

correct 

Time 

% 

used 

60 

IF000478 

5C 

Data 

Real 

Percent change 

2 

50% 

85 

54% 

81 

30% 

61 

DM001370 

5C 

Data 

Real 

Percent change 

2 

47% 

87 

45% 

82 

30% 

62 

DM001394 

QC 

Data 

Real 

Mean/median Comparison 

1 

39% 

72 

41% 

69 

29% 

63 

WW002249 

QC 

Arithmetic 

Pure 

Exponents 

-1 

56% 

31 

57% 

34 

29% 

64 

LB006128 

QC 

Arithmetic 

Real 

Ratio, proportion 

2 

59% 

76 

60% 

72 

28% 

65 

DW002092 

5C 

Data 

Real 

How many categories 

2 

76% 

73 

77% 

71 

28% 

66 

WW002073 

5C 

Arithmetic 

Real 

Rate (e.g., MPG, dollars/hour) 

1 

74% 

84 

74% 

84 

28% 

67 

DC000083 

QC 

Algebra 

Pure 

Linear inequality 

2 

63% 

58 

63% 

57 

27% 

68 

M-035486 

5C 

Data 

Real 

Ratio, proportion 

2 

38% 

108 

37% 

107 

27% 

69 

DW000520 

QC 

Arithmetic 

Real 

Measurement conversion 

2 

71% 

58 

73% 

56 

26% 

70 

DM001369 

5C 

Data 

Real 

Ratio, proportion 

1 

76% 

71 

74% 

72 

26% 

71 

AY000912 

QC 

Algebra 

Pure 

Linear inequality 

2 

48% 

64 

55% 

63 

26% 

72 

DM100219 

5C 

Algebra 

Real 

Ratio, proportion 

1 

75% 

74 

77% 

69 

25% 

73 

DM002196 

5C 

Data 

Real 

Miscellaneous 

2 

38% 

100 

39% 

92 

24% 

74 

BE000402 

QC 

Algebra 

Pure 

Positive and negative numbers 

0 

70% 

34 

67% 

40 

24% 


(Table continues) 



Table (continued) 


Item classification No calculator Calculator available 


Item 

number 

Accession 

number 

Type 

1 

2 

Subject 

Developer 

rating 

% 

correct 

Time 

% 

correct 

Time 

% 

used 

75 

DQ003175 

5C 

Arithmetic 

Real 

Graduated rate 

1 

72% 

80 

70% 

77 

24% 

76 

DM001891 

QC 

Data 

Real 

Counting (combinatorics) 

1 

40% 

80 

41% 

80 

22% 

77 

VB110055 

5C 

Algebra 

Pure 

Series and sequences 

0 

30% 

79 

32% 

78 

21% 

78 

WW002063 

QC 

Geometry 

Pure 

Triangle area 

1 

69% 

51 

67% 

58 

21% 

79 

DM002300 

5C 

Data 

Pure 

Median 

1 

78% 

50 

79% 

54 

21% 

80 

LB010349 

5C 

Data 

Real 

Read data 

2 

56% 

88 

53% 

90 

20% 

81 

DM001717 

5C 

Data 

Real 

Ratio, proportion 

2 

53% 

110 

56% 

114 

20% 

82 

VB340313 

QC 

Data 

Real 

Weighted mean 

1 

62% 

67 

66% 

71 

19% 

83 

UB100012 

QC 

Geometry 

Pure 

Combination 

2 

41% 

91 

41% 

86 

19% 

84 

LB010436 

5C 

Data 

Real 

C omputation—fractions 

1 

90% 

61 

89% 

60 

19% 

85 

DM001645 

QC 

Arithmetic 

Pure 

Negative exponents 

1 

62% 

40 

60% 

38 

19% 

86 

VB383223 

QC 

Algebra 

Real 

Percent of percent 

2 

41% 

82 

40% 

81 

18% 

87 

DM001714 

5C 

Data 

Real 

Computation—integers 

1 

56% 

124 

58% 

123 

18% 

88 

LB002848 

QC 

Geometry 

Pure 

Pythagorean theorem 

2 

33% 

49 

35% 

49 

18% 

89 

VB363670 

QC 

Data 

Pure 

Median 

0 

62% 

52 

62% 53 16% 

(Table continues) 



Table (continued) 


Item classification No calculator Calculator available 


Item 

number 

Accession 

number 

Type 

1 

2 

Subject 

Developer 

rating 

% 

correct 

Time 

% 

correct 

Time 

% 

used 

90 

UB100001 

QC 

Data 

Real 

Percent change 

2 

59% 

63 

56% 

61 

16% 

91 

VB355893 

QC 

Geometry 

Pure 

Combination: 

2 

73% 

36 

72% 

36 

16% 

92 

VB106018 

5C 

Arithmetic 

Real 

Series and sequences 

1 

46% 

98 

46% 

94 

16% 

93 

AY002228 

QC 

Algebra 

Pure 

Systems of equations/inequalities 

1 

42% 

102 

39% 

96 

16% 

94 

DM001169 

5C 

Geometry 

Pure 

Combination 

1 

23% 

94 

25% 

92 

16% 

95 

PV000271 

QC 

Algebra 

Pure 

Algebraic manipulation 

1 

55% 

55 

54% 

57 

16% 

96 

DM001623 

QC 

Geometry 

Real 

Circle area 

2 

66% 

60 

65% 

58 

16% 

97 

SE002197 

QC 

Arithmetic 

Real 

Percent change 

2 

66% 

40 

65% 

41 

15% 

98 

M-077877 

QC 

Algebra 

Pure 

Computation—radicals 

-1 

40% 

39 

41% 

41 

14% 

99 

LB001446 

QC 

Arithmetic 

Pure 

Properties of operations 

1 

55% 

42 

56% 

42 

14% 

100 

DM001642 

5C 

Algebra 

Pure 

Systems of equations/inequalities 

1 

71% 

83 

71% 

83 

13% 

101 

VB348669 

5C 

Data 

Pure 

Counting (combinatorics) 

0 

29% 

63 

30% 

69 

13% 

102 

VB366165 

5C 

Arithmetic 

Pure 

Negative exponents 

1 

57% 

45 

59% 

44 

12% 

103 

SE002196 

QC 

Geometry 

Pure 

Combination 

1 

67% 

55 

67% 

54 

12% 

104 

VB328701 

QC 

Geometry 

Pure 

Quadrilateral area 

1 

75% 

46 

75% 

45 

12% 
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Table (continued) 


Item classification No-calculator Calculator-available 


Item 

number 

Accession 

number 

Type 

1 

2 

Subject 

Developer 

rating 

% 

correct 

Time 

% 

correct 

Time 

% 

used 

105 

LB010364 

QC 

Arithmetic 

Pure 

Properties of operations 

2 

69% 

32 

73% 

35 

11% 

106 

DQ000381 

QC 

Geometry 

Pure 

Quadrilateral area 

1 

55% 

49 

54% 

49 

11% 

107 

LB010276 

QC 

Algebra 

Pure 

Newly defined functions 

1 

83% 

37 

82% 

37 

11% 

108 

AY001323 

QC 

Algebra 

Pure 

Algebraic manipulation 

0 

72% 

58 

74% 

60 

11% 

109 

WW002167 

QC 

Arithmetic 

Real 

Computation—integers 

1 

87% 

46 

87% 

45 

11% 

110 

VB384970 

5C 

Arithmetic 

Pure 

Series and sequences 

2 

38% 

79 

40% 

78 

11% 

111 

DM100273 

QC 

Algebra 

Real 

Rate (e.g., MPG, dollars/hour) 

1 

61% 

66 

57% 

70 

10% 

112 

LB007923 

5C 

Arithmetic 

Real 

Rate (e.g., MPG, dollars/hour) 

1 

89% 

47 

91% 

43 

10% 

113 

VB382618 

QC 

Arithmetic 

Pure 

Remainders 

0 

36% 

53 

37% 

52 

9% 

114 

VB352058 

QC 

Arithmetic 

Real 

Graduated rate 

1 

87% 

43 

88% 

43 

9% 

115 

VB179819 

QC 

Algebra 

Pure 

Computation—fractions 

0 

61% 

66 

56% 

66 

9% 

116 

WV000243 

5C 

Data 

Pure 

Sets 

1 

82% 

55 

81% 

52 

8% 

117 

DQ001593 

QC 

Data 

Pure 

Mean/median comparison 

1 

64% 

42 

64% 

42 

8% 

118 

VB368885 

5C 

Data 

Pure 

Probability 

1 

28% 

87 

30% 

87 

7% 

119 

DC030101 

QC 

Geometry 

Pure 

Combination: 

1 

65% 

49 

65% 

47 

7% 
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Table (continued) 


Item 

number 

Accession 

number 




Item classification 


No calculator 

Calculator available 

Type 

1 

2 

Subject 

Developer 

rating 

% 

correct 

Time 

% 

correct 

Time 

% 

used 

120 

LB010346 

5C 

Data 

Real 

Which category/categories 

2 

34% 

51 

34% 

51 

7% 

121 

VB155808 

QC 

Arithmetic 

Pure 

Series and sequences 

0 

41% 

83 

39% 

90 

7% 

122 

AY001121 

QC 

Geometry 

Pure 

Angles in the plane 

0 

70% 

54 

66% 

53 

7% 

123 

DM002163 

QC 

Geometry 

Pure 

Coordinate geometry 

0 

47% 

40 

46% 

40 

7% 

124 

VB376019 

QC 

Algebra 

Pure 

Quadratic/other equation/inequality 

0 

52% 

67 

52% 

65 

6% 

125 

DW001388 

5C 

Data 

Real 

Percent (basic) 

1 

52% 

76 

51% 

71 

6% 

126 

WV000104 

QC 

Geometry 

Pure 

Triangle area 

0 

84% 

32 

82% 

33 

6% 

127 

DC000518 

5C 

Geometry 

Pure 

Triangle area 

1 

72% 

51 

72% 

50 

6% 

128 

DM001844 

QC 

Data 

Pure 

Probability 

1 

59% 

38 

63% 

34 

6% 

129 

DM001363 

QC 

Algebra 

Pure 

Coordinate geometry 

2 

29% 

54 

31% 

55 

6% 

130 

DC000276 

QC 

Arithmetic 

Pure 

Computation—fractions 

0 

61% 

38 

58% 

36 

6% 

131 

DQ007318 

QC 

Arithmetic 

Real 

Rate (e.g., MPG, dollars/hour) 

0 

75% 

53 

77% 

52 

6% 

132 

VB352654 

QC 

Geometry 

Pure 

Triangle perimeter/triangle inequality 

1 

30% 

68 

29% 

72 

6% 

133 

VB315299 

QC 

Algebra 

Pure 

Algebraic manipulation 

-1 

59% 

60 

63% 

58 

6% 

134 

VB352042 

QC 

Data 

Real 

Sets 

1 

37% 

47 

37% 

43 

5% 
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Table (continued) 


Item classification No calculator Calculator available 


Item 

number 

Accession 

number 

Type 

1 

2 

Subject 

Developer 

rating 

% 

correct 

Time 

% 

correct 

Time 

% 

used 

135 

DR000516 

QC 

Geometry 

Pure 

Coordinate geometry 

1 

67% 

51 

69% 

52 

5% 

136 

JM001367 

QC 

Algebra 

Pure 

Systems of equations/inequalities 

0 

90% 

31 

89% 

31 

5% 

137 

DC000487 

5C 

Data 

Real 

Order 

1 

77% 

72 

76% 

70 

5% 

138 

WW001069 

QC 

Algebra 

Pure 

Properties of operations 

0 

77% 

31 

79% 

31 

5% 

139 

WW001426 

QC 

Algebra 

Pure 

Computation—integers 

1 

87% 

26 

88% 

26 

5% 

140 

LB009196 

5C 

Algebra 

Pure 

Absolute value 

0 

48% 

72 

45% 

70 

4% 

141 

VB333458 

QC 

Algebra 

Pure 

Slope 

1 

78% 

75 

75% 

75 

4% 

142 

VB143721 

QC 

Arithmetic 

Pure 

Positive and negative numbers 

1 

87% 

32 

89% 

33 

3% 

143 

VB384454 

QC 

Algebra 

Pure 

Negative exponents 

0 

36% 

43 

33% 

43 

3% 

144 

DT000175 

QC 

Algebra 

Pure 

Algebraic manipulation 

0 

40% 

58 

40% 

56 

3% 

145 

VB378850 

5C 

Arithmetic 

Pure 

Factors, multiples, divisibility 

0 

70% 

67 

65% 

65 

3% 

146 

LB002943 

QC 

Geometry 

Pure 

Lines and segments 

1 

63% 

54 

61% 

51 

3% 

147 

DQ001806 

5C 

Data 

Pure 

Counting (combinatorics) 

0 

84% 

50 

81% 

48 

2% 

148 

LB008785 

5C 

Algebra 

Pure 

Absolute value 

0 

54% 

44 

55% 

42 

2% 

149 

DM001790 

QC 

Data 

Pure 

Probability 

0 

30% 

22 

31% 

22 

2% 
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Table (continued) 


Item classification No calculator Calculator available 


Item 

number 

Accession 

number 

Type 

1 

2 

Subject 

Developer 

rating 

% 

correct 

Time 

% 

correct 

Time 

% 

used 

150 

DC000153 

5C 

Algebra 

Pure 

Algebraic manipulation 

0 

67% 

61 

61% 

57 

2% 

151 

DM001220 

5C 

Algebra 

Pure 

Positive and negative numbers 

0 

72% 

44 

71% 

43 

2% 

152 

M-075074 

QC 

Arithmetic 

Real 

Ratio, proportion 

0 

53% 

58 

52% 

56 

2% 

153 

NT000450 

QC 

Algebra 

Pure 

Quadratic/other equation/inequality 

0 

49% 

59 

46% 

56 

2% 

154 

DQ001633 

QC 

Arithmetic 

Pure 

Primes, prime factorization 

1 

74% 

38 

76% 

38 

2% 

155 

AY002477 

QC 

Geometry 

Pure 

Angles in the plane 

0 

69% 

47 

68% 

48 

2% 

156 

WW002335 

5C 

Geometry 

Pure 

Combination: circle/quad/area/peri 

1 

85% 

38 

86% 

37 

2% 

157 

UB100529 

QC 

Algebra 

Pure 

Newly defined functions 

1 

47% 

49 

47% 

46 

2% 

158 

WW002340 

QC 

Arithmetic 

Pure 

Factors, multiples, divisibility 

1 

65% 

28 

63% 

27 

1% 

159 

DQ002900 

QC 

Algebra 

Pure 

Order 

1 

74% 

32 

73% 

30 

1% 

160 

LB002905 

5C 

Arithmetic 

Pure 

Exponents 

0 

37% 

55 

35% 

54 

1% 

161 

LB001664 

QC 

Geometry 

Real 

Quadrilateral area 

0 

52% 

60 

49% 

58 

1% 

162 

VB358467 

5C 

Geometry 

Pure 

Coordinate geometry 

0 

48% 

85 

48% 

85 

1% 

163 

VB324359 

QC 

Algebra 

Pure 

Positive and negative numbers 

1 

26% 

32 

31% 

32 

1% 

164 

DM001942 

5C 

Data 

Pure 

Standard deviation 

0 

39% 

31 

40% 

31 

1% 


(Table continues) 




Table (continued) 


Item 

number 

Accession 

number 




Item classification 


No calculator 

Calculator available 

Type 

1 

2 

Subject 

Developer 

rating 

% 

correct 

Time 

% 

orrect 

Time 

% 

used 

165 

LB022237 

5C 

Data 

Real 

Read data 

1 

33% 

66 

30% 

64 

1% 

166 

AY002342 

QC 

Geometry 

Pure 

Triangle perimeter/triangle inequality 

0 

42% 

37 

42% 

36 

1% 

167 

DM001628 

5C 

Geometry 

Pure 

X, Y intercepts 


47% 

63 

47% 

63 

0% 

168 

DC005082 

QC 

Geometry 

Pure 

Combination 

0 

85% 

20 

84% 

20 

0% 



Appendix B 
Item Examples 


Item 48 


These questions refer to the following table. 


Population Profile of the United States, Census Years 1900 - 1990 


Year 

Population 

Percent 

increase 

Population per 
square mile 

Percent 

urban/rural 

Median age 
(in years) 

1900 

75,994,575 

20.7 

25.6 

39.6/60.4 

22.9 

1910 

91,972,266 

21.0 

31.0 

45.6/54.4 

24.1 

1920 

105,710,620 

14.9 

35.6 

51.2/48.8 

25.3 

1930 

122,775,046 

16.1 

41.2 

56.1/43.9 

26.4 

1940 

131,669,275 


44.2 

56.5/43.5 

29.0 

1950 

150,697,361 

14.5 

50.7 

64.0/36.0 

30.2 

1960 

179,323,175 

18.5 

50.6 

69.9/30.1 

29.5 

1970 

203,302,031 

13.4 

57.4 

73.5/26.5 

28.0 

1980 

226,545,805 

11.4 

64.0 

73.7/26.3 

30.0 

1990 

NA 

248,709,873 
= Not available 

9.8 

NA 

NA 

33.0 


For how many of the census years from 1900 to 1980 was the ration of the urban population to 
the rural population 

greater than 7 to 4? O 0ne O Two O Three O Four O Five 


% correct: 

No-Calculator = 34 
Calculator-Available = 43 

36% in calculator-available group used the calculator 
% correct: 

Available, not used = 27 
Available, used = 71 


Mean GRE-Q Score: 
Available, not used = 567 
Available, used = 657 


This difficult item became substantially easier with a calculator, but did not become 
trivial with calculator availability. Even with an available calculator, fewer than half of the 
examinees got this item correct. 
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Item 5 

These questions refer to the following graphs. 

Distribution of Funds By Charity X 

1992 



The increase in the amount of money 
distributed for family support from 
1992 to 1993 was closest to which of 
the following? 


o 

$0 

o 

$24,000 

o 

$40,000 

o 

$60,000 

o 

$94,000 


Total Funds Distributed: $2.54 million 


1993 



Services to 
People with 
Disabilities 


% correct: 

No-calculator = 49 
Calculator-available = 64 

57% in calculator-available group 
used the calculator 

% correct: 

Available, not used = 43 
Available, used = 81 


Mean GRE-Q Score: 
Available, not used =578 
Available, used = 619 


This calculation-intensive item is of middle-difficulty and becomes considerably 
easier, but not trivially easy, with calculator availability. 
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Item 8 


Pat purchased 4 large picture frames for $20.00 each, 100 medium-sized 
frames for $10.00 each, and 8 small frames for $5.00 each. What was the 
average price per picture frame? 


o 

$7.50 

o 

$10.00 

o 

$11.67 

o 

$15.00 

o 

$17.00 


% correct: 

No-calculator = 75 
Calculator-available = 79 

56% in calculator-available group used the calculator 

% correct: Mean GRE-Q Score: 

available, not used = 72 available, not used = 600 

available, used = 85 available, used = 597 


This easy item becomes only slightly easier with calculator availability. Although it 
initially appears that a calculator might be useful, the calculations actually required are so 
simple that the calculator is of little use. 
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